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TECHNICAL FIELD 



[0002] The described technology stabilizes an image stream created by an 

airborne video camera. 

BACKGROUND 

[0003] If not stabilized, the image streams created by airborne video cameras can 

be practically unusable for human observation because frame-to-frame image 
jitter is excessive. This image jitter typically is caused by small, fast pointing 
errors superposed upon larger-amplitude, slower pointing errors. 

[0004] It is possible to reduce this image jitter to acceptable levels with refined 

mechanical stabilization techniques, stabilizing the line of sight of the image so 
that image jitter amplitude is less than an acceptable limit. Such an approach can 
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deliver high-quality image streams from all types of cameras (video or film) but 
leads to large, heavy mechanical systems for support of the airborne camera. 
Such systems are the subject of U.S. Patent Nos. 5,897,223; 3,638,502; 
4,989,466; 4,643,539; and 5,184,521. An approach relying purely upon 
mechanical stabilization leads to heavy and complex mechanical systems. 
Usually, multiple nested mechanical stages of stabilization are required, with each 
stage reducing the image jitter further, purely by reduction in jitter of the line of 
sight. 

[0005] It would be desirable to have a technique to reduce jitter and avoid the 

need to have such large, heavy, and expensive mechanical systems. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0006] Figure 1 is a block diagram that illustrates the bases of the stabilization 

system in one embodiment. 
[0007] Figure 2 is a block diagram that illustrates nesting of the two types of image 

stabilization in one embodiment. 
[0008] Figure 3 is a block diagram that illustrates a coupling from the camera 

pointing system to the stabilization system in one embodiment. 
[0009] Figure 4A is a graph that illustrates a typical error budget for the 

cooperative efforts of the various levels of image stabilization employed by the 

stabilization system in one embodiment. 
[0010] Figure 4B is a block diagram illustrating two types of electronic stabilization 

systems. 

[0011] Figure 5 is a diagram illustrating the reference frames used in the 

electromechanical stabilization system. 
[0012] Figure 6 illustrates the pixel offsets on an object from one frame to the next 

in one embodiment. 

[0013] Figure 7 is a flow diagram illustrating the processing of the stabilization 

system in one embodiment. 
[0014] Figure 8 is a flow diagram illustrating the processing of the analyze pixel 

offsets for velocity component in one embodiment. 
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[0015] Figure 9 is a flow diagram illustrating the processing of the adjust pixel 

offsets for aircraft velocity in one embodiment. 
[0016] Figure 10 is a flow diagram illustrating the processing of the adjust pixel 

offsets for camera rotation component in one embodiment. 
[0017] Figure 11 is a flow diagram illustrating the processing of the adjust for 

image analysis by the electromechanical stabilization in one embodiment. 

DETAILED DESCRIPTION 

[0018] A method and system for stabilizing images being taken by a video camera 

using electromechanical stabilization is provided. In one embodiment, the 
stabilization system performs inter-frame stabilization based on the velocity of a 
vehicle on which the video camera is mounted and the pan rate of a line-of-sight 
controller of the video camera. The inter-frame stabilization is performed by a 
software component by moving a display area (or viewport) within a larger image 
area. The inter-frame stabilization removes small-amplitude jitter while accounting 
for vehicle velocity and orientation, pan rate and orientation of the line-of-sight 
controller, distance to an object within the image, and field of view of the camera. 
The stabilization system converts an inter-frame stabilization adjustment into a 
pan rate adjustment so the line-of-sight controller will keep the desired object 
within the image area of the camera. In this way, the stabilization system uses an 
electronic stabilization to remove small-amplitude jitters and feeds those 
adjustments to a mechanical stabilization to account for large-amplitude jitter. 

[0019] In one embodiment, the stabilization system comprises a video camera 

controlled by a gimbal-based, line-of-sight controller that is mounted on an aircraft. 
While the aircraft is flying, the video camera feeds images to the software 
component that provides the inter-frame stabilization based on the scan and tilt 
rate (i.e., pan rate) of the line-of-sight controller. The software component 
removes small-amplitude jitter while factoring in the scan and tilt rate of the line-of- 
sight controller. The software component receives images from the camera that 
are larger than the display area. The software component moves the display area 
around within the larger image to remove the small-amplitude jitter. The software 
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component then calculates a scan and tilt rate adjustment for the line-of-sight 
controller. The software component then provides the adjustment to the line-of- 
site controller so it can keep the video camera at the desired line of sight. 
[0020] The stabilization system nests mechanical stabilization and electronic 

stabilization loops to exploit modern capabilities in electronics and image 
processing. Because not all of the stabilization is achieved mechanically, a 
simpler, cheaper, smaller, lighter, and lower-power mechanical gimbal system 
may be used. 

[0021] The stabilization system uses an electronic image stabilization to augment 

mechanical line-of-sight stabilization to achieve full frame-to-frame stabilization of 
the image flow. The mechanical system is used for the large-amplitude, slow line- 
of-sight corrections required, while electronic stabilization is used for the small- 
amplitude, faster corrections not handled by the mechanical system. These 
stabilization loops are nested to take advantage of the characteristics of both 
types of stabilization. The stabilization system can implement various levels of 
interaction between these stabilization methods. 

Inner-to-Outer Nulling 

[0022] The fastest, smallest-amplitude stabilization is implemented electronically 

by "sliding" successive frames in the image stream on the display screen or on the 
focal plane array within the camera. This type of stabilization accounts for small 
amplitudes, typically a small fraction of the frame. The stabilization system 
provides the image correction implemented by this electronic stabilization to the 
mechanical pointing system (i.e., the line-of-sight controller) so that the 
mechanical pointing system can implement movements to cause the long-term 
average electronic image correction to tend toward zero. If such corrections are 
not implemented by the mechanical pointing system, then the displayed image 
might slowly drift and exceed the limits of practical correction of the electronic 
stabilization. 
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Outer-to-lnn r Coupling 

[0023] A user may want the image to "flow" across the screen, as for example, 

when the camera is panned while images are being gathered. An electronic 
stabilization system may misinterpret such image flow as unwanted jitter and will 
attempt to correct for it. Such misinterpretation would lead to momentarily 
stabilized images with sudden "steps" required when the electronic correction 
reaches its practical limit. The stabilization system can prevent such sudden 
steps if provided with image flow from the command system of the mechanical 
pointing system. Thus, the stabilization system can be used to enable smooth 
electronic image stabilization, even when the camera is being panned across a 
scene and the image flows across the display screen. 

[0024] Figure 1 is a block diagram that illustrates the bases of the stabilization 

system in one embodiment. Large-amplitude stabilization of camera line of sight 
is implemented with a motor-driven gimbal system 101 for controlling a camera 
105. Small-amplitude stabilization of inter-frame motion is accomplished by 
electronic stabilization techniques implemented via an image processor 102. 
These electronic techniques shift the displayed image on a display screen 103 
electronically by an adjustment and feed the adjustment to a gimbal controller 

104. The image processor 102 may include a central processing unit, memory, 
input devices (e.g., keyboard and pointing devices), output devices (e.g., display 
devices), and storage devices (e.g., disk drives). The memory and storage 
devices are computer-readable media that may contain instructions that 
implement the electronic stabilization of the video stream provided by the camera 

105. In addition, data structures and message structures (e.g., gimbal controller 
command and gimbal scan and tilt rate) may be stored or transmitted via a data 
transmission medium, such as a signal on a communications link. 

[0025] Figure 2 is a block diagram that illustrates the adjustment of the pan rate of 

the camera based on the inter-frame adjustment in one embodiment. The 
adjustment to the mechanical pointing system reduces any accumulating 
electronic image re-registration by mechanically re-pointing the camera. This 
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allows the stabilization system to stabilize the image stream with a minimal loss of 
image size. 

[0026] Figure 3 is a block diagram that illustrates the adjustment of the inter-frame 

stabilization based on the pan rate of the camera in one embodiment. This 
adjustment enables the stabilization system to stabilize successive images not 
simply in the display frame but rather to stabilize only unwanted motion between 
successive images. Thus, when the camera line of sight is intentionally panning 
the scene, the desired image flow is not suppressed by the stabilization system. 
Rather, the stabilization system suppresses only deviations from this desired 
image flow. 

[0027] Figure 4A is a graph that illustrates a typical error budget for the 

cooperative efforts of the various levels of image stabilization employed by the 
stabilization system in one embodiment. Low-speed, large-amplitude stabilization 
is implemented by the mechanical gimbal system, while high-speed, small- 
amplitude stabilization is implemented by the electronic stabilization system. 

[0028] Figure 4B is a block diagram illustrating two types of electronic stabilization 

systems. Gyro-to-image electronic stabilization 401 measures mechanical motion 
of the camera body and implements image re-registration to compensate for this 
mechanical motion. Such electronic image stabilization is implemented in some 
hand-held "camcorder" video cameras sold in retail stores by Sony, Panasonic, 
and others. Gyro-to-image electronic stabilization is most practically implemented 
in airborne hardware and software within an airborne camera system. Frame-to- 
frame electronic image stabilization 402 employs image recognition techniques to 
re-register every frame prior to its display in a succession of image frames. 
Frame-to-frame stabilization attempts to minimize unwanted shift of successive 
frames on the display screen. 

[0029] Figure 5 is a diagram illustrating the reference frames used in the 

stabilization system in one embodiment. The reference frame of the earth is 
represented by north N, east E, and down D coordinates. The position of an 
aircraft, which may be provided by an altimeter and a GPS system, is in the earth 
reference frame. The reference frame of the body of the aircraft is represented by 

[36761-801 6-US01 00/SL03321 0. 1 50] -6- 1 1/25/03 



heading B1, pitch B2, and roll B3 coordinates that may be provided by the 
aircraft's gyros. The reference frame of the camera is represented by a line of 
sight C1, tilt C2, and scan C3 coordinates. In one embodiment, the camera is 
controlled by an inertial stabilization system that controls the gimbal motors to 
control the orientation of the C1, C2, and C3 camera axes. The 
electromechanical stabilization system receives camera scan and tilt rate 
information from the camera rate gyros and adjusts these rates to further account 
for frame-to-frame jitter information and image recognition, which are provided by 
the electronic stabilization. 
[0030] The stabilization system inputs the images generated by the camera, the 

velocity of the aircraft in the earth reference frame V a j rcraft , the camera scan rate 

and tilt rate, the orientations of the aircraft and the camera, and the distance to an 
object in the images. The stabilization system analyzes consecutive frames and 
determines the optimal translation of one frame to make it best coincide with the 
preceding frame. The stabilization system may use conventional pattern 
recognition techniques to locate the object within the image. The stabilization 
system provides an offset in pixels to best superimpose one frame onto the next 
frame. The pixel offsets may be represented as the number of pixels horizontally 
(also referred to as the scan direction) and the number of pixels vertically (also 
referred to as the tilt direction) on the display. In one embodiment, the 
stabilization system has an image buffer in memory that is larger than the 
displayed image. When the stabilization system detects variations in the position 
of an object resulting from jitter, it can offset the displayed image by the calculated 
scan and tilt offset, providing a frame that best superimposes with the previous 
displayed frame, thus effectively removing the jitter. 
[0031] Figure 6 illustrates the pixel offsets on an object from one frame to the next 

in one embodiment. The memory buffer stores the image received from the 
camera using 2000 by 2000 pixels. However, only 1800 by 1800 pixels are 
displayed on the display. Thus, the image can be adjusted by 100 pixels in the 
negative and positive scan and tilt directions to account for jitter, assuming the last 
frame was centered in the memory buffer. In this example, a car is at position 

[36761 -801 6-US0100/SL03321 0.1 50] -7- 11/25/03 



(1000, 1000) in the first frame, and the upper left corner of the display 
corresponds to position (101, 101). In a subsequent frame, the car is now at 
position (1010, 1020). Thus, the stabilization system can display position (111, 
121) as the upper left corner of the display to place the car at the same position 
on the display from one frame to the next frame. 

[0032] Since the camera may be panning a scene and the aircraft platform may be 

moving relative to the scene, a portion of the pixel offsets calculated by the 
stabilization system may be a result of this desired movement. In such a case, 
the stabilization system is provided with aircraft velocity and orientation, camera 
line of sight and orientation, and camera scan and tilt rate to estimate and factor 
out this desired movement before adjusting the image. The stabilization system 
calculates the sum of the pixel offsets resulting from the aircraft velocity and 
orientation, and the camera orientation and angular rate. The stabilization system 
then subtracts this sum from the pixel offsets calculated from the image analysis 
to give the pixel offsets attributable to the jitter. 

[0033] . Because the number of offset pixels is limited, the electromechanical 

stabilization loop keeps the desired image in the center of the camera. To do so, 
, the stabilization system uses the pixel offsets to re-center the gimbal angles of the 
camera. The stabilization system converts the pixel offsets to corresponding scan 
and tilt error. The stabilization system also calculates scan and tilt rates. It then 
adjusts the scan and tilt rate of the camera to track an object or prevent an 
overflow of the pixel offset in the stabilization system. 

[0034] The stabilization system uses transformation matrices to represent the 

current orientation of the body of the aircraft relative to the earth reference frame 
and the current orientation of the camera to the body reference frame. The 
camera reference frame relative to the body of the plane reference frame is 
represented by a transformation matrix C C b for transforming a vector from the 
body reference frame to the camera reference frame. C C b is a 3-by-3 matrix 
whose columns are orthogonal and normalized, also referred to as a matrix of 
direction cosines. The following equation represents the conversion of a position 
in the body reference frame to the camera reference frame: 
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R C =C CB R B (1) 

where R B represents the position in the body reference frame and R c represents 
the position in the camera reference frame. An example C C b is 



2-1/2 


_2"1/2 
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The matrix C C b is set based on the angles of the gimbal relative to the body. 
Thus, this matrix represents the current gimbal angles. A matrix C B e is for 
transforming from the earth reference frame to the body reference frame. Thus, 
the matrix C B e represents the heading, pitch, and roll of the aircraft as measured 
by the gyro of the aircraft. 
[0035] Figure 7 is a flow diagram illustrating the processing of the stabilization 

system in one embodiment. The stabilization system calculates an initial image 
adjustment based on analysis of the image. The stabilization system then adjusts 
the initial adjustment based on velocity and orientation of the aircraft and the scan 
and tilt rate of the camera. In block 701, the system invokes a component to 
analyze the image and calculate image pixel offsets. The invoked component 
provides the scan and tilt pixel offsets. The component may use standard object 
recognition techniques to recognize an object in successive frames and set the 
initial adjustment based on the difference in locations of the object. In block 702, 
the system invokes the adjust pixel offset component, which determines the 
velocity pixel offsets that are attributable to the desired aircraft velocity, the 
rotation of the camera, and the orientations of the aircraft and the camera. In 
blocks 703-704, the system calculates pixel offsets due to unwanted image jitter 
by subtracting the velocity pixel offsets and desired camera rotation pixel offsets 
from the image pixel offsets. In block 705, the stabilization system displays the 
stabilized image by selecting the portion of the screen that begins at the newly 
calculated offsets. In block 706, the system invokes the adjust for image analysis 
component providing the pixel offsets to calculate a change in the scan and tilt 
rate of the camera to help ensure that the image correction will not exceed the 
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maximum number of allowable pixels. In block 707, the system adjusts the scan 
and tilt rate of the camera. The system may be invoked to process every frame or 
only a subset of the frames (e.g., every 10 frames). 
[0036] Figure 8 is a flow diagram illustrating the processing of the analyze pixel 

offsets for velocity component in one embodiment. The component is provided 
with the velocity of the aircraft V a j rcraft in the earth reference frame, the orientation 

of the aircraft C B e, and the orientation of the camera C C b. In block 801, the 
component invokes the adjust pixel offsets for aircraft velocity component to 
calculate the pixel offsets attributable to the velocity of the aircraft. In block 802, 
the component invokes the adjust pixel offsets for camera rotation component to 
calculate the pixel offsets attributable to the rotation of the camera. In blocks 803- 
804, the component combines the pixel offsets for the scan and tilt directions. 
[0037] Figure 9 is a flow diagram illustrating the processing of the adjust pixel 

offsets for aircraft velocity in one embodiment. In block 901, the component 
calculates a transformation matrix for transforming from the earth reference frame 
to the camera reference frame as follows: 

Cce = C C bC B e (3) 

In block 902, the component calculates the line of sight of the camera in the earth 
reference frame as 

L E =C CE T (1A0) T (4) 

where L E is the line of sight of the camera in the earth reference frame and where 
the superscript T represents the transpose of the matrix or vector. In block 903, 
the component retrieves the distance or range K to the object at the center of the 
camera. The range may be provided by a range finder or by calculating the 
distance using the altitude of the target. For example, if the object is at sea level, 
then the distance can be calculated based on the altitude of the aircraft and the 
angle of the line of sight. In block 904, the component transforms the velocity of 
the aircraft to the camera reference frame as 
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V c =C*V E (5) 
In block 905, the component normalizes the velocity of the aircraft as 

Vl raft =V^ rcraft /K (6) 

where V? renift is the normalized velocity of the aircraft in radians per hour. For 

example, if the velocity of the aircraft in the scan direction is 100km/hr and the 
distance to the object is 1km, then the normalized velocity is 100rad/hr, which 
means the aircraft moves in the scan direction 100 times the distance to the object 
in one hour. In block 906, the component calculates the difference in scan units 
as 

AS c =Vl raft (S)*AT (7) 

where AT is the frame refresh period. For example, when the normalized velocity 
is 100rad/hr and the refresh rate is 15 times per second, then the change in scan 
units is: 

lOOrad \hr Xsec 1 , 

* * = rad - .1 \degrees (8) 

hr 3600sec 15 540 

In block 907, the component calculates the aircraft pixel offset in the scan 
direction by converting the difference in scan units to the corresponding pixel 
offset factoring in the field of view (or zoom) of the camera. The component 
calculates the pixel offset as 

APO(S)=AS c *P/Z (9) 

where APO(S) is the pixel offset in the scan direction, Z is the zoom factor, and P 
is the pixel density. For example, if the scan units are 1/540rad and there are 
2000 pixels in the scan direction with a field of view of .93rad (1km field of view at 
1km distance), the pixel offset is 

Xrad 2000 pixels , . , 

* = 4 pixels (10) 

540 .93rad 
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In blocks 908-909, the component calculates the pixel offset in the tilt direction in 
a similar manner. 

[0038] Figure 10 is a flow diagram illustrating the processing of the adjust pixel 

offsets for the camera rotation component in one embodiment. The component 
receives the instantaneous camera scan and tilt rates from the gimbal controller. 
Alternatively, the component can calculate them based on the orientations of the 
aircraft and the camera. In block 1001, the component calculates the difference in 
scan units as 

AS C =IS*AT (11) 

where IS is the instantaneous scan rate of the camera measured by a rate gyro. 
In block 1002, the component calculates the difference in tilt units as 

AT C = IT*AT (12) 

where IT is the instantaneous tilt rate of the camera measured by a rate gyro. In 
block 1003, the component calculates the camera pixel offset in the scan direction 
by converting the difference in scan units to the corresponding pixel offset, 
factoring in the field of view (or zoom) of the camera. In block 1004, the 
component calculates the pixel offset in the tilt direction in a similar manner. 
[0039] Figure 11 is a flow diagram illustrating the processing of the adjust for 

image analysis by the stabilization in one embodiment. The component is passed 
the pixel offsets corresponding to the adjustments made by the stabilization 
system. In blocks 1101-1102, the component converts the pixel offsets to 
corresponding scan angle in radians and tilt angle in radians, factoring in the field 
of view of the camera. In blocks 1 103-1 104, the component calculates the angle 
error resulting from the pixel offsets as 

AE(S) = AS C *K (13) 
AE(T) = AT C *K (14) 

where AE is the angle error and K is the distance to the center of the image. In 
blocks 1105-1106, the component calculates the adjustments for the angle errors 
as 
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A(S) = (W/AT)*AE(S) (15) 
A(T) = (W/AT)*AE(T) (16) 

where A(S) is the adjustment for the scan rate in radians per second and W is a 
weighting factor that controls the bandwidth of the feedback loop. The weighting 
factor controls the speed at which adjustments can be made to the scan and tilt 
rates. The stabilization system compares the adjustment to the scan rate of the 
camera provided by the gyro and uses the difference in rate to control the velocity 
of the gimbal motors. 

[0040] All patents and articles cited herein are hereby incorporated by reference in 

their entirety and relied upon. Further details of methods of operating airborne 
cameras in accordance with other embodiments of the invention are described in 
U.S. Patent Application No. 60/440,983, filed on January 17, 2003, and entitled, 
"Compensation for Overflight Velocity When Stabilizing an Airborne Camera," and 
U.S. Patent Application No. 60/440,977, filed on January 17, 2003, and entitled, 
"Method and Apparatus for Stabilizing Payload, Including Airborne Cameras," 
which are hereby incorporated by reference. 

[0041] One skilled in the art will appreciate that although specific embodiments of 

the stabilization system have been described herein for purposes of illustration, 
various modifications may be made without deviating from the spirit and scope of 
the invention. For example, the principles of the stabilization system may be used 
on a transport mechanism other than an airplane, such as a satellite, a rocket, a 
missile, a train, an automobile, and so on. In addition, the camera may even be 
stationary or not traveling relative to an object in the video. Accordingly, the 
invention is not limited except by the appended claims. 
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